Working with Test Configurations

A test configuration is a combination of test cases and exploratory tests that can be run on demand or be scheduled to run on a regular basis in Eggplant DAI. A test configuration is essentially a blueprint of tests that can be run one after the other against a system under test (SUT) in a specific testing sequence.

Go to Controller > Test Config, to create, edit, or view your defined test configurations. The following columns are displayed to provide at-a-glance details about each test configuration:

• Name: Display name of the test configuration.

• Model/Suite: Name of the model or suite depending on the type of test configuration you've chosen to run.

• Description: Description of the test configuration. Provide a meaningful description to help distinguish it from other test configurations.

• Tags: Tags defined for the selected test configuration that you're viewing.

• SUT: Associated SUT, or criteria for choosing a SUT for the test configuration.

• Updated: Date and time when the test configuration was last updated.

• Steps: Number of test steps included in the test configuration. A test step is either a test case from the model or a script based test case, or an exploratory test with stop criteria, which could be defined by duration, iterations, or a target coverage percentage.

Click the Actions menu to Edit or Delete a test configuration.

To filter test configurations, type part of a test configuration name in the Search field or if you know the name of the model, select it from the All Models drop-down list.

Creating Test Configurations

Go to Controller > Test Config to create tests that can be run on a regular basis. You must specify at least one SUT against which the test will be run and associate it with a model.

note

For exploratory steps, a stop criterion is always set by duration, iterations, or target coverage. After the test runs out of steps, Eggplant DAI will examine the execution of each step and indicate whether the test configuration is a success or failure.

Step by Step: Creating a Test Configuration

Follow these steps to add a test configuration:

1. Go to Controller > Test Config.

2. From the top menu, click Add New +.

3. Enter the Name and Description for the test configuration so that you can easily identify it in your existing list of test configurations.

4. Add Tags by entering a name for the test configuration tag and pressing Enter. You can add as many test configuration tags as you need.

5. Specify the SUT selection for the test to run on by selecting the appropriate options:

   • None SUT if the SUT is defined in your code or not required.
   • SUT by name to select a predefined SUT.
   • SUT by criteria tag to define one or more SUT criteria from the list. All the SUTs that are defined on the SUTs tab are available in this drop-down list. The SUT criteria field is similar to Test configuration tags, i.e., if you enter a SUT criteria and press Enter, relevant values are populated.

   For Cloud users, the options are slightly different:

   • Execution Environment: Select from the options below to determine whether you want to use DAI Eggplant Cloud Execution Environment including the Agent and EPF with Cloud SUTs or your local SUTs defined in the SUTs tab. Select Cloud to execute tests on Cloud SUTs. For using the Cloud SUT in Design mode, see Connecting to DAI Cloud SUTs.

   

   • Local: Lets you select a predefined SUT or define a SUT based on criteria. To define a SUT based on criteria, enter a SUT criteria and press Enter.

   • Cloud: Enables the Eggplant Cloud Hosted SUT option:

     • Eggplant hosted cloud: Select this to execute tests on Cloud SUTs where Eggplant will provision the Cloud Execution Environment (including the Agent and Eggplant Functional) for you. For more information, see Connecting to DAI Cloud SUTs.

     To execute tests using Cloud Execution Environment and your own local SUTs, you must ensure that your firewall rules allow connections from the Cloud Execution Environment. All connections from the Cloud Execution Environment to the SUTs will come from one of these fixed IP addresses:

     • 54.195.73.56
     • 54.170.198.89
     • 54.170.92.66

6. Select the Type of test configuration you want to run. The selection you make here will determine the steps to add to your test configuration.

   

   • Script Based: Enables the option to select a suite and then add script based steps. Note that scripts are only be available if you have at least associated one suite in your Model tab. For detailed steps, see Using Script Based Test Configuration.
   • Model Based: Enables the option to select a model in order to run model based test configuration.

7. For Model Based test configuration, select a model, then add an exploratory step or directed test cases, or both.

   • To add an exploratory step, select Add Exploratory and specify a target for the exploratory test steps by selecting one of the following options:
     • Iterations: Determines the number of times that the exploratory test should execute before moving on to next step in the test configuration. The test configuration will end if there is no other step left in the list of tests. By default, the iteration value is set to 10.
     • Coverage: Determines the minimum coverage required to complete a step. In Eggplant DAI, coverage represents the percentage of states and actions visited in your model during a series of exploratory test runs. Note: A test will pass even when the coverage doesn’t meet the minimum target coverage. For e.g., the test will stop when the target coverage is not achieved within a reasonable time, or until the point at which coverage stops increasing.
     • Time: Select to specify the duration of the exploratory step in hours.
   • To add specific directed test cases to your test configuration, select from the list. Note that test cases will only appear if you have created at least one directed test case in the Test Cases tab in Modeler.

8. Click Save. The new test configuration is saved to the list of test configurations in the Test Config tab.

Tips

• Reorder your steps by clicking the Drag icon, then drag and drop steps as required.
• Delete your steps by clicking the Delete icon.

Enabling Reruns

The rerun feature allows you to save time and effort by automatically rerunning failed and errored test steps up to 50 times without executing the entire test configuration. A test step can be a model based test case, a script based test case, or an exploratory test.

To activate the rerun functionality for failed or errored test steps:

1. Enable reruns by sliding the Automatically rerun failed and errored test steps toggle to the right. (A green check indicates the option is enabled.)

2. Configure the rerun parameters with the following controls:

   • Number of Runs: Specify the maximum rerun count for failed and errored test steps. You can adjust this value using the text box or slider, with a maximum of 50 reruns.
   • Wait Duration: Define the interval between each rerun, optimizing the rerun process. Choose a time interval from the dropdown list, measured in seconds. The default is set to zero, implying no wait interval.

This streamlined process allows you to effectively manage reruns, enhancing your testing workflow.

note

• The chosen rerun count and wait duration will be saved even if you deactivate the rerun feature. This way, you can easily re-enable it at a later time without reconfiguring.
• Access rerun statistics by viewing the total reruns completed under the Controller menu on the Runner page. Expand individual test results to find this information.

Running a Test Configuration

You can run a test configuration from Controller > Runner by clicking against the test you want to run.

Go to Controller > Runner to configure test schedules with specific frequencies to be run at regular intervals. You can only have one schedule per test configuration.

tip

Remember to always associate suites to a model before running tests.

note

When you abort a test run that is part of a test configuration schedule, all runs that are part of that configuration schedule are aborted.

Eggplant Functional Stop and Restart Settings for Test Configuration Runs

When running a test configuration, you can configure the length of time to run Eggplant Functional (EPF) before restarting or the grace periods for poll-time stopping and restarting with the following environment variables:

Environment Variables	Default	Description
RESTART_EPF_AFTER_SECONDS	1800	The period from when EPF launches until the next EPF restart. This variable takes effect after the current step finishes and before moving to the next. It only applies to test configurations run with the DAI Run Agent. During live runs, EPF runs continuously without restarting. Default: 30 minutes.
EPF_QUIT_POLL_TIME_SECONDS	20	The grace period for EPF to quit during the restart process.
EPF_RESTART_POLL_TIME_SECONDS	120	The grace period for EPF to start up after having quit. EPF_RESTART_POLL_TIME_SECONDS does not include EPF_QUIT_POLL_TIME_SECONDS.

Viewing, Editing and Notification Settings of an Existing Test Configuration

To view or edit a test configuration:

1. Go to Controller > Test Config.
2. Find the test configuration that you're interested in updating or viewing, then from the Actions menu, select Edit.
3. From here you can view or update the test configuration details.
4. Furthermore, you can also manage the notification settings of the test configuration by clicking on link.

Notification Settings of an Existing Test Configuration

Important

Test Configuration notifications must be enabled globally before you can set up notifications for a specific Test Configuration within DAI. You need to be an administrator to do this. See Manage DAI Settings for details.

By default, the notification feature for all Test Configurations is disabled. To enable it, follow these steps:

1. Go to Controller > Test Config.
2. Find the test configuration that you're interested in, then from the Actions menu, select Edit.
3. Click on link.
4. From here, you can enable / disable the feature and assign verified users to receive the notifications.
5. Besides, you can also manage the Notification Type you wish to receive by selecting from the various options.
6. These are the rules for Recipients:
   • If the feature is enabled, then you have to pick at least one recipient before you are allowed to save.
   • If the feature is disabled, then you can leave the field blank and save.
7. These are the rules for Notification Type:
   • If the feature is enabled and you uncheck all checkboxes, the feature will toggle to the disabled status.
   • If the feature is disabled and all checkboxes are unchecked, when you toggle to enable the feature, all checkboxes will be checked by default.

Removing a Test Configuration

To remove an existing test configuration:

1. Go to Controller > Test Config.
2. Find the test configuration that you want to delete, then from the Actions menu, select Delete.
3. From here you can view or update the test configuration details.

Duplicating an Existing Test Configuration

To duplicate an existing test configuration:

1. Go to Controller > Test Config.
2. Find the test configuration that you want to duplicate, then from the Actions menu, select Duplicate.
3. From here, you will be required to enter a new name for the Test Config.
4. Click Create once a valid name is entered.
5. From here, you will be able to edit your newly-duplicated Test Config.

Tips

• Make sure to avoid entering the same name as the Test Config that is to be duplicated.
• The new name has to follow these rules:
  1. 2 to 100 alphanumeric characters.
  2. Space and dashes are accepted, however they cannot be used at the start or end of the name, only between other alphanumeric values.
  3. No other special characters are allowed.
• Examples of acceptable names:
  1. New Test Config
  2. New-Test-Config
  3. New-Test
  4. New Test 2